Learn TensorFlow Enterprise by KC Tung

Author:KC Tung [KC Tung]
Language: eng
Format: epub
Publisher: Packt Publishing
Published: 2020-11-26T16:00:00+00:00

Using the Google Cloud GPU through AI Platform

Having worked through the previous section for utilizing Cloud TPU with AI Platform, we are ready to do the same with the GPU. As it turns out, the formats of training script and invocation commands are very similar. With the exception of a few more parameters and slight differences in the distributed strategy definition, everything else remains the same.

There are several distributed strategies (https://www.tensorflow.org/guide/distributed_training#types_of_strategies) currently available. For a TensorFlow Enterprise distribution in Google AI Platform, MirroredStrategy and TPUStrategy are the only two that are fully supported. All the others are experimental. Therefore, in this section's example, we will use MirroredStrategy. This strategy creates copies of all the variables in the model on each GPU. As these variables are updated at each gradient decent step, the value updates are copied to each GPU synchronously. By default, this strategy uses an NVIDIA NCCL all-reduce implementation. Now, we are going to start with the following steps:

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.